HIVE-3002: IBMCloud MachinePools: Snowflake FailureDomain matching #2825

2uasimojo · 2026-01-06T23:24:17Z

We've been using cluster-control-plane-machineset-operator (CPMS) utilities to extract failure domains from MachineSets for comparison when matching up generated msets with remote ones. CPMS doesn't support IBMCloud, so we were quietly getting back a generic (empty) failure domain for all IBMCloud msets, resulting in false positive matches and bad controller behavior (namely deleting all but one of the remote MachineSets).

Here we add local code to interpret the Zone from the providerSpec as the failure domain and match on it.

openshift-ci-robot · 2026-01-06T23:24:22Z

@2uasimojo: This pull request references HIVE-3002 which is a valid jira issue.

Details

In response to this:

We've been using cluster-control-plane-machineset-operator (CPMS) utilities to extract failure domains from MachineSets for comparison when matching up generated msets with remote ones. CPMS doesn't support IBMCloud, so we were quietly getting back a generic (empty) failure domain for all IBMCloud msets, resulting in false positive matches and bad controller behavior (namely deleting all but one of the remote MachineSets).

Here we add local code to interpret the Zone from the providerSpec as the failure domain and match on it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

2uasimojo · 2026-01-06T23:24:29Z

/hold for QE

openshift-ci · 2026-01-06T23:24:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2uasimojo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [2uasimojo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2026-01-07T01:13:33Z

Codecov Report

❌ Patch coverage is 46.15385% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.70%. Comparing base (26db049) to head (3f207cb).
⚠️ Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
...g/controller/machinepool/machinepool_controller.go	46.15%	3 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2825      +/-   ##
==========================================
+ Coverage   50.40%   50.70%   +0.30%     
==========================================
  Files         279      279              
  Lines       34194    34534     +340     
==========================================
+ Hits        17236    17511     +275     
- Misses      15597    15642      +45     
- Partials     1361     1381      +20

Files with missing lines	Coverage Δ
...g/controller/machinepool/machinepool_controller.go	`67.00% <46.15%> (+3.77%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

huangmingxia · 2026-01-14T20:20:07Z

/test konflux

openshift-ci · 2026-01-14T20:20:14Z

@huangmingxia: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test coverage

/test e2e

/test e2e-azure

/test e2e-gcp

/test e2e-openstack

/test e2e-pool

/test e2e-vsphere

/test images

/test periodic-images

/test security

/test unit

/test verify

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-hive-master-coverage

pull-ci-openshift-hive-master-e2e

pull-ci-openshift-hive-master-e2e-pool

pull-ci-openshift-hive-master-images

pull-ci-openshift-hive-master-periodic-images

pull-ci-openshift-hive-master-security

pull-ci-openshift-hive-master-unit

pull-ci-openshift-hive-master-verify

Details

In response to this:

/test konflux

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

huangmingxia · 2026-01-14T20:39:23Z

/retest

red-hat-konflux · 2026-01-14T20:39:33Z

Caution

There are some errors in your PipelineRun template.

PipelineRun	Error
hive-mce-210-on-pull-request	`CEL expression evaluation error: expression "event == \"pull_request\"\n&& !body.pull_request.draft\n&& target_branch == \"master\"\n&& !files.all.all(x, x.matches('^docs/\|\\\\.md$\|^(?:.*/)?(?:\\\\.gitignore\|OWNERS\|PROJECT\|LICENSE)$'))\n" failed to evaluate: no such key: pull_request`
hive-mce-211-on-pull-request	`CEL expression evaluation error: expression "event == \"pull_request\"\n&& !body.pull_request.draft\n&& target_branch == \"master\"\n&& !files.all.all(x, x.matches('^docs/\|\\\\.md$\|^(?:.*/)?(?:\\\\.gitignore\|OWNERS\|PROJECT\|LICENSE)$'))\n" failed to evaluate: no such key: pull_request`
hive-mce-26-on-pull-request	`CEL expression evaluation error: expression "event == \"pull_request\"\n&& !body.pull_request.draft\n&& target_branch == \"master\"\n&& !files.all.all(x, x.matches('^docs/\|\\\\.md$\|^(?:.*/)?(?:\\\\.gitignore\|OWNERS\|PROJECT\|LICENSE)$'))\n" failed to evaluate: no such key: pull_request`
hive-mce-27-on-pull-request	`CEL expression evaluation error: expression "event == \"pull_request\"\n&& !body.pull_request.draft\n&& target_branch == \"master\"\n&& !files.all.all(x, x.matches('^docs/\|\\\\.md$\|^(?:.*/)?(?:\\\\.gitignore\|OWNERS\|PROJECT\|LICENSE)$'))\n" failed to evaluate: no such key: pull_request`
hive-mce-28-on-pull-request	`CEL expression evaluation error: expression "event == \"pull_request\"\n&& !body.pull_request.draft\n&& target_branch == \"master\"\n&& !files.all.all(x, x.matches('^docs/\|\\\\.md$\|^(?:.*/)?(?:\\\\.gitignore\|OWNERS\|PROJECT\|LICENSE)$'))\n" failed to evaluate: no such key: pull_request`
hive-mce-29-on-pull-request	`CEL expression evaluation error: expression "event == \"pull_request\"\n&& !body.pull_request.draft\n&& target_branch == \"master\"\n&& !files.all.all(x, x.matches('^docs/\|\\\\.md$\|^(?:.*/)?(?:\\\\.gitignore\|OWNERS\|PROJECT\|LICENSE)$'))\n" failed to evaluate: no such key: pull_request`
hive-on-pull-request	`CEL expression evaluation error: expression "event == \"pull_request\"\n&& !body.pull_request.draft\n&& target_branch == \"master\"\n&& !files.all.all(x, x.matches('^docs/\|\\\\.md$\|^(?:.*/)?(?:\\\\.gitignore\|OWNERS\|PROJECT\|LICENSE)$'))\n" failed to evaluate: no such key: pull_request`

pkg/controller/machinepool/machinepool_controller.go

We've been using cluster-control-plane-machineset-operator (CPMS) utilities to extract failure domains from MachineSets for comparison when matching up generated msets with remote ones. CPMS doesn't support IBMCloud, so we were quietly getting back a generic (empty) failure domain for all IBMCloud msets, resulting in false positive matches and bad controller behavior (namely deleting all but one of the remote MachineSets). Here we add local code to interpret the `Zone` from the providerSpec as the failure domain and match on it.

2uasimojo · 2026-01-26T19:55:54Z

/test security

2uasimojo · 2026-01-26T19:56:45Z

/retest hive-mce-28

openshift-ci-robot · 2026-01-26T19:58:14Z

@2uasimojo: This pull request references HIVE-3002 which is a valid jira issue.

Details

In response to this:

We've been using cluster-control-plane-machineset-operator (CPMS) utilities to extract failure domains from MachineSets for comparison when matching up generated msets with remote ones. CPMS doesn't support IBMCloud, so we were quietly getting back a generic (empty) failure domain for all IBMCloud msets, resulting in false positive matches and bad controller behavior (namely deleting all but one of the remote MachineSets).

Here we add local code to interpret the Zone from the providerSpec as the failure domain and match on it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

2uasimojo · 2026-01-26T19:58:57Z

/test e2e

2uasimojo · 2026-01-26T19:59:12Z

/test hive-mce-26-on-pull-request

openshift-ci · 2026-01-26T22:01:15Z

@2uasimojo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/verify	`3f207cb`	link	true	`/test verify`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 6, 2026

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 6, 2026

openshift-ci bot requested review from dlom and suhanime January 6, 2026 23:24

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 6, 2026

huangmingxia reviewed Jan 19, 2026

View reviewed changes

pkg/controller/machinepool/machinepool_controller.go Outdated Show resolved Hide resolved

2uasimojo force-pushed the HIVE-3002/ibmcloud-fd-matching branch from 86596dd to 3f207cb Compare January 19, 2026 16:58

openshift deleted a comment from openshift-ci bot Jan 26, 2026

2uasimojo closed this Jan 26, 2026

2uasimojo reopened this Jan 26, 2026

openshift deleted a comment from openshift-ci bot Jan 26, 2026

HIVE-3002: IBMCloud MachinePools: Snowflake FailureDomain matching #2825

Are you sure you want to change the base?

HIVE-3002: IBMCloud MachinePools: Snowflake FailureDomain matching #2825

Uh oh!

Conversation

2uasimojo commented Jan 6, 2026

Uh oh!

openshift-ci-robot commented Jan 6, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

2uasimojo commented Jan 6, 2026

Uh oh!

openshift-ci bot commented Jan 6, 2026

Uh oh!

codecov bot commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

huangmingxia commented Jan 14, 2026

Uh oh!

openshift-ci bot commented Jan 14, 2026

Uh oh!

huangmingxia commented Jan 14, 2026

Uh oh!

red-hat-konflux bot commented Jan 14, 2026

Uh oh!

Uh oh!

2uasimojo commented Jan 26, 2026

Uh oh!

2uasimojo commented Jan 26, 2026

Uh oh!

openshift-ci-robot commented Jan 26, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

2uasimojo commented Jan 26, 2026

Uh oh!

2uasimojo commented Jan 26, 2026

Uh oh!

openshift-ci bot commented Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

openshift-ci-robot commented Jan 6, 2026 •

edited by openshift-ci bot

Loading

codecov bot commented Jan 7, 2026 •

edited

Loading

openshift-ci-robot commented Jan 26, 2026 •

edited by openshift-ci bot

Loading